{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "849dbd89-ce04-4a18-84fb-c19f3db5504a",
   "metadata": {},
   "source": [
    "# Building optimized RAG with LlamaIndex + DSPy\n",
    "\n",
    "This notebook provides a comprehensive overview of LlamaIndex + DSPy integrations.\n",
    "\n",
    "We show **three** core integrations:\n",
    "1. **Build and optimize Query Pipelines with DSPy predictors**: The first section shows you how to write DSPy code to define signatures for LLM inputs/outputs. Then port over these components to overall workflows within LlamaIndex Query pipelines, and then end-to-end optimize the entire system.\n",
    "\n",
    "2. **Build and optimize Query Pipelines with Existing Prompts**: Instead of writing DSPy signatures, you can just define a LlamaIndex prompt template, and our converter will auto-optimize it for you.\n",
    "\n",
    "3. **Port over DSPy-Optimized Prompts to any LlamaIndex Module**: Possible through our `DSPyPromptTemplate` - translate an optimized prompt through DSPy into any module that requires prompts in LlamaIndex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa558b8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index==0.10.44"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d223313-af5b-4155-8896-c24aa4cb6925",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Define the LLM setting for DSPy (note: this is separate from using the LlamaIndex LLMs), and also the answer signature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c5c50d5e-7046-40c5-bf3e-a8d2a2a2c6f8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import dspy\n",
    "\n",
    "turbo = dspy.OpenAI(model='gpt-3.5-turbo')\n",
    "dspy.settings.configure(lm=turbo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ac68be4f-36b1-4054-99dd-707ce42f61a4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import dspy\n",
    "\n",
    "class GenerateAnswer(dspy.Signature):\n",
    "    \"\"\"Answer questions with short factoid answers.\"\"\"\n",
    "\n",
    "    context_str = dspy.InputField(desc=\"contains relevant facts\")\n",
    "    query_str = dspy.InputField()\n",
    "    answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eec2a981-2ee1-497a-b675-15e1029183f6",
   "metadata": {},
   "source": [
    "## [Part 1] Build and Optimize a Query Pipeline with DSPy Modules\n",
    "\n",
    "Use our DSPy query components to plugin DSPy prompts/LLMs, stitch together with our query pipeline abstraction.\n",
    "\n",
    "Any query pipeline can be plugged into our `LlamaIndexModule`. We can then let DSPy optimize the entire thing e2e."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "113f0637-993f-4bef-bd8a-1122951cf7f8",
   "metadata": {},
   "source": [
    "#### Load Data, Build Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "55c9d434-c8f6-403e-9499-36059eb09907",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-06-17 23:54:09--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8002::154, 2606:50c0:8000::154, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 75042 (73K) [text/plain]\n",
      "Saving to: ‘paul_graham_essay.txt’\n",
      "\n",
      "paul_graham_essay.t 100%[===================>]  73.28K  --.-KB/s    in 0.01s   \n",
      "\n",
      "2024-06-17 23:54:10 (7.48 MB/s) - ‘paul_graham_essay.txt’ saved [75042/75042]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# port it over to another index  (paul graham example) \n",
    "\n",
    "!wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt -O paul_graham_essay.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "367a7985-2f89-47be-b2af-14fef2e845c9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core import SimpleDirectoryReader, VectorStoreIndex\n",
    "\n",
    "reader = SimpleDirectoryReader(input_files=[\"paul_graham_essay.txt\"])\n",
    "docs = reader.load_data()\n",
    "\n",
    "index = VectorStoreIndex.from_documents(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c9a68920-6517-465b-8c0c-4c6e0b8390a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "retriever = index.as_retriever(similarity_top_k=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8d43c6c-9dee-4cbc-b39d-5d8ad617b6ea",
   "metadata": {},
   "source": [
    "#### Build Query Pipeline\n",
    "\n",
    "Replace the synthesis piece with the DSPy component (make sure GenerateAnswer matches signature of inputs/outputs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ac7df48b-9369-4f17-8542-0f8afd9e621e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.query_pipeline import QueryPipeline as QP, InputComponent, FnComponent\n",
    "from dspy.predict.llamaindex import DSPyComponent, LlamaIndexModule\n",
    "\n",
    "dspy_component = DSPyComponent(\n",
    "    dspy.ChainOfThought(GenerateAnswer)\n",
    ")\n",
    "\n",
    "retriever_post = FnComponent(\n",
    "    lambda contexts: \"\\n\\n\".join([n.get_content() for n in contexts])\n",
    ")\n",
    "\n",
    "\n",
    "p = QP(verbose=True)\n",
    "p.add_modules(\n",
    "    {\n",
    "        \"input\": InputComponent(),\n",
    "        \"retriever\": retriever,\n",
    "        \"retriever_post\": retriever_post,\n",
    "        \"synthesizer\": dspy_component,\n",
    "    }\n",
    ")\n",
    "p.add_link(\"input\", \"retriever\")\n",
    "p.add_link(\"retriever\", \"retriever_post\")\n",
    "p.add_link(\"input\", \"synthesizer\", dest_key=\"query_str\")\n",
    "p.add_link(\"retriever_post\", \"synthesizer\", dest_key=\"context_str\")\n",
    "\n",
    "\n",
    "dspy_qp = LlamaIndexModule(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b457f1db-3315-476c-8673-bf19ad6e1657",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;2;155;135;227m> Running module input with input: \n",
      "query_str: what did the author do in YC\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever with input: \n",
      "input: what did the author do in YC\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever_post with input: \n",
      "contexts: [NodeWithScore(node=TextNode(id_='49a290f5-5f29-413c-97e7-9fdf15169cf4', embedding=None, metadata={'file_path': 'paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain'...\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module synthesizer with input: \n",
      "query_str: what did the author do in YC\n",
      "context_str: YC was different from other kinds of work I've done. Instead of deciding for myself what to work on, the problems came to me. Every 6 months there was a new batch of startups, and their problems, what...\n",
      "\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "output = dspy_qp(query_str=\"what did the author do in YC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7b78501f-e281-4e68-bdb6-1eeca6bf2464",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(\n",
       "    answer='Worked with startups, funded them.'\n",
       ")"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "691d5d86-1ce0-46fd-9fbc-eca68043c935",
   "metadata": {},
   "source": [
    "#### Optimize Query Pipeline\n",
    "\n",
    "Let's try optimizing the query pipeline with few-shot examples.\n",
    "\n",
    "We define a toy dataset with two examples. We then use our `SemanticSimilarityEvaluator` to define a custom eval function to pass to the DSPy teleprompter.\n",
    "- Because our passing threshold is set to very low, every example should pass with a reasonable LLM. \n",
    "- What this practically means is that all training examples will be added as few-shot examples to the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ec3a783c-2025-4b1c-9f5b-cdcde7211ace",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from dspy import Example\n",
    "\n",
    "train_examples = [\n",
    "    Example(query_str=\"What did the author do growing up?\", answer=\"The author wrote short stories and also worked on programming.\"),\n",
    "    Example(query_str=\"What did the author do during his time at YC?\", answer=\"organizing a Summer Founders Program, funding startups, writing essays, working on a new version of Arc, creating Hacker News, and developing internal software for YC\")\n",
    "]\n",
    "\n",
    "train_examples = [t.with_inputs(\"query_str\") for t in train_examples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2e43a18c-5569-4947-a02d-08f52331b134",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "502cc403-8ebc-4bbf-be7c-c15385909b7a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|                                                                        | 0/2 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;2;155;135;227m> Running module input with input: \n",
      "query_str: What did the author do growing up?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever with input: \n",
      "input: What did the author do growing up?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever_post with input: \n",
      "contexts: [NodeWithScore(node=TextNode(id_='20a73a69-f604-450e-b07d-cace28b471a5', embedding=None, metadata={'file_path': 'paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain'...\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module synthesizer with input: \n",
      "query_str: What did the author do growing up?\n",
      "context_str: What I Worked On\n",
      "\n",
      "February 2021\n",
      "\n",
      "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to...\n",
      "\n",
      "\u001b[0m"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 50%|████████████████████████████████                                | 1/2 [00:00<00:00,  1.85it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;2;155;135;227m> Running module input with input: \n",
      "query_str: What did the author do during his time at YC?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever with input: \n",
      "input: What did the author do during his time at YC?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever_post with input: \n",
      "contexts: [NodeWithScore(node=TextNode(id_='49a290f5-5f29-413c-97e7-9fdf15169cf4', embedding=None, metadata={'file_path': 'paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain'...\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module synthesizer with input: \n",
      "query_str: What did the author do during his time at YC?\n",
      "context_str: YC was different from other kinds of work I've done. Instead of deciding for myself what to work on, the problems came to me. Every 6 months there was a new batch of startups, and their problems, what...\n",
      "\n",
      "\u001b[0m"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  2.07it/s]\n"
     ]
    }
   ],
   "source": [
    "from dspy.teleprompt import BootstrapFewShot\n",
    "from llama_index.core.evaluation import SemanticSimilarityEvaluator\n",
    "\n",
    "evaluator = SemanticSimilarityEvaluator(similarity_threshold=0.5)\n",
    "\n",
    "# Validation logic: check that the predicted answer is correct.\n",
    "# Also check that the retrieved context does actually contain that answer.\n",
    "def validate_context_and_answer(example, pred, trace=None):\n",
    "    result = evaluator.evaluate(response=pred.answer, reference=example.answer)\n",
    "    return result.passing\n",
    "\n",
    "# Set up a basic teleprompter, which will compile our RAG program.\n",
    "teleprompter = BootstrapFewShot(max_labeled_demos=0, metric=validate_context_and_answer)\n",
    "\n",
    "# Compile!\n",
    "compiled_dspy_qp = teleprompter.compile(dspy_qp, trainset=train_examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3a147c3e-febe-43fa-bca5-152432d3662b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;2;155;135;227m> Running module input with input: \n",
      "query_str: How did PG meet Jessica Livingston?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever with input: \n",
      "input: How did PG meet Jessica Livingston?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever_post with input: \n",
      "contexts: [NodeWithScore(node=TextNode(id_='465a8143-6740-4bbd-8e6c-bb5eba4bb5a3', embedding=None, metadata={'file_path': 'paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain'...\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module synthesizer with input: \n",
      "query_str: How did PG meet Jessica Livingston?\n",
      "context_str: Over the next several years I wrote lots of essays about all kinds of different topics. O'Reilly reprinted a collection of them as a book, called Hackers & Painters after one of the essays in it. I al...\n",
      "\n",
      "\u001b[0m"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(\n",
       "    answer='Met at a party in 2003.'\n",
       ")"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test this out \n",
    "compiled_dspy_qp(query_str=\"How did PG meet Jessica Livingston?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96a2ba4e-6677-44e7-a474-db382ad1a56a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# [optional]: inspect history\n",
    "turbo.inspect_history(n=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9895ceff-5aeb-4285-8ede-99bb35a50a45",
   "metadata": {},
   "source": [
    "## [Part 2] Build and Optimize Query Pipelines with Existing Prompts\n",
    "\n",
    "Build a query pipeline similar to the previous section. But instead of directly using DSPy signatures/predictors, we can build DSPyComponent modules from LlamaIndex prompts directly. \n",
    "\n",
    "This allows you to write any LlamaIndex prompt and trust that it'll be optimized in DSPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8efff3a9-77d9-4a89-a0f7-207d63e73ab0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.prompts import PromptTemplate\n",
    "\n",
    "# let's try a fun prompt that writes in Shakespeare! \n",
    "qa_prompt_template = PromptTemplate(\"\"\"\\\n",
    "Context information is below.\n",
    "---------------------\n",
    "{context_str}\n",
    "---------------------\n",
    "Given the context information and not prior knowledge, \\\n",
    "answer the query.\n",
    "\n",
    "Write in the style of a Shakespearean sonnet.\n",
    "\n",
    "Query: {query_str}\n",
    "Answer: \n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "a84aa7b2-b9b8-4740-b862-9073470c31c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.query_pipeline import QueryPipeline as QP, InputComponent, FnComponent\n",
    "from dspy.predict.llamaindex import DSPyComponent, LlamaIndexModule\n",
    "\n",
    "dspy_component = DSPyComponent.from_prompt(qa_prompt_template)\n",
    "\n",
    "retriever_post = FnComponent(\n",
    "    lambda contexts: \"\\n\\n\".join([n.get_content() for n in contexts])\n",
    ")\n",
    "\n",
    "\n",
    "p = QP(verbose=True)\n",
    "p.add_modules(\n",
    "    {\n",
    "        \"input\": InputComponent(),\n",
    "        \"retriever\": retriever,\n",
    "        \"retriever_post\": retriever_post,\n",
    "        \"synthesizer\": dspy_component,\n",
    "    }\n",
    ")\n",
    "p.add_link(\"input\", \"retriever\")\n",
    "p.add_link(\"retriever\", \"retriever_post\")\n",
    "p.add_link(\"input\", \"synthesizer\", dest_key=\"query_str\")\n",
    "p.add_link(\"retriever_post\", \"synthesizer\", dest_key=\"context_str\")\n",
    "\n",
    "\n",
    "dspy_qp = LlamaIndexModule(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "e23b7662-cb87-4906-bbaf-e2ffd74e20d8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StringSignature(context_str, query_str -> sonnet_answer\n",
       "    instructions='Essential Instructions: Provide an answer to the query based solely on the context information provided. The response should be written in the style of a Shakespearean sonnet, which typically consists of 14 lines written in iambic pentameter, with a rhyme scheme of ABABCDCDEFEFGG.'\n",
       "    context_str = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context Str:', 'desc': '${context_str}'})\n",
       "    query_str = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Query Str:', 'desc': '${query_str}'})\n",
       "    sonnet_answer = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Sonnet Answer:', 'desc': '${sonnet_answer}'})\n",
       ")"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the inferred signature\n",
    "dspy_component.predict_module.signature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "6044855f-cdd2-43f0-9d2e-e7ba64a28e27",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|                                                                        | 0/2 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;2;155;135;227m> Running module input with input: \n",
      "query_str: What did the author do growing up?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever with input: \n",
      "input: What did the author do growing up?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever_post with input: \n",
      "contexts: [NodeWithScore(node=TextNode(id_='20a73a69-f604-450e-b07d-cace28b471a5', embedding=None, metadata={'file_path': 'paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain'...\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module synthesizer with input: \n",
      "query_str: What did the author do growing up?\n",
      "context_str: What I Worked On\n",
      "\n",
      "February 2021\n",
      "\n",
      "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to...\n",
      "\n",
      "\u001b[0m"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 50%|████████████████████████████████                                | 1/2 [00:00<00:00,  1.94it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;2;155;135;227m> Running module input with input: \n",
      "query_str: What did the author do during his time at YC?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever with input: \n",
      "input: What did the author do during his time at YC?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever_post with input: \n",
      "contexts: [NodeWithScore(node=TextNode(id_='49a290f5-5f29-413c-97e7-9fdf15169cf4', embedding=None, metadata={'file_path': 'paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain'...\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module synthesizer with input: \n",
      "query_str: What did the author do during his time at YC?\n",
      "context_str: YC was different from other kinds of work I've done. Instead of deciding for myself what to work on, the problems came to me. Every 6 months there was a new batch of startups, and their problems, what...\n",
      "\n",
      "\u001b[0m"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.95it/s]\n"
     ]
    }
   ],
   "source": [
    "from dspy.teleprompt import BootstrapFewShot\n",
    "from llama_index.core.evaluation import SemanticSimilarityEvaluator\n",
    "from dspy import Example\n",
    "\n",
    "output_key = \"sonnet_answer\"\n",
    "train_example_dicts = [\n",
    "    {\"query_str\": \"What did the author do growing up?\", output_key: \"The author wrote short stories and also worked on programming.\"},\n",
    "    {\"query_str\": \"What did the author do during his time at YC?\", output_key: \"organizing a Summer Founders Program, funding startups, writing essays, working on a new version of Arc, creating Hacker News, and developing internal software for YC\"}\n",
    "]\n",
    "train_examples = [Example(**t).with_inputs(\"query_str\") for t in train_example_dicts]\n",
    "\n",
    "evaluator = SemanticSimilarityEvaluator(similarity_threshold=0.5)\n",
    "# Validation logic: check that the predicted answer is correct.\n",
    "# Also check that the retrieved context does actually contain that answer.\n",
    "def validate_context_and_answer(example, pred, trace=None):\n",
    "    result = evaluator.evaluate(response=getattr(pred, output_key), reference=getattr(example, output_key))\n",
    "    return result.passing\n",
    "\n",
    "# Set up a basic teleprompter, which will compile our RAG program.\n",
    "teleprompter = BootstrapFewShot(max_labeled_demos=0, metric=validate_context_and_answer)\n",
    "\n",
    "# Compile!\n",
    "compiled_dspy_qp = teleprompter.compile(dspy_qp, trainset=train_examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "16fcb28c-c442-4f1d-8876-9c6c6c47eda0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;2;155;135;227m> Running module input with input: \n",
      "query_str: How did PG meet Jessica Livingston?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever with input: \n",
      "input: How did PG meet Jessica Livingston?\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module retriever_post with input: \n",
      "contexts: [NodeWithScore(node=TextNode(id_='465a8143-6740-4bbd-8e6c-bb5eba4bb5a3', embedding=None, metadata={'file_path': 'paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain'...\n",
      "\n",
      "\u001b[0m\u001b[1;3;38;2;155;135;227m> Running module synthesizer with input: \n",
      "query_str: How did PG meet Jessica Livingston?\n",
      "context_str: Over the next several years I wrote lots of essays about all kinds of different topics. O'Reilly reprinted a collection of them as a book, called Hackers & Painters after one of the essays in it. I al...\n",
      "\n",
      "\u001b[0m"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Prediction(\n",
       "    sonnet_answer=\"In the midst of a party, bright and gay,\\nA clever scheme brought guests together, true.\\nAmong them, Jessica, in a charming way,\\nCaught the author's eye, a friendship grew.\\n\\nShe, a marketer in a bank of old,\\nDiscovered startup tales, colorful and bold.\\nAs the bank faced troubles, she sought anew,\\nVenture capital's flaws came into view.\\n\\nTheir paths converged on a fateful night,\\nAt the corner of Garden and Walker streets.\\nA decision made, a future bright,\\nTo start an investment firm, their feats.\\n\\nThus, through ignorance and boldness, they began,\\nA journey in angel investing, a novel plan.\"\n",
       ")"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test this out \n",
    "compiled_dspy_qp(query_str=\"How did PG meet Jessica Livingston?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2b1b5f7-66d0-49d2-9bae-ca94f12a124d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# [optional]: inspect the optimized prompt \n",
    "turbo.inspect_history(n=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4daae77-bf36-418f-9a80-c9c0fb6f20ec",
   "metadata": {},
   "source": [
    "## [Part 3] Port over Optimized Prompts to LlamaIndex using the DSPy Prompt Template\n",
    "\n",
    "Extract out a prompt from an existing compiled DSPy module, and then port it over to any LlamaIndex pipeline! \n",
    "\n",
    "In the example below we use our `DSPyPromptTemplate` to extract out the compiled few-shot prompt from the optimized query pipeline. \n",
    "\n",
    "We then plug it into a separate query engine over the PG essay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "8180ea2b-412a-41a6-a707-f2945d9dcda0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from dspy.predict.llamaindex import DSPyPromptTemplate\n",
    "\n",
    "# NOTE: you cannot do DSPyPromptTemplate(dspy_component.predict_module) - the predict_module is replaced.\n",
    "qa_prompt_tmpl = DSPyPromptTemplate(compiled_dspy_qp.query_pipeline.module_dict[\"synthesizer\"].predict_module)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "e3c86e63-f225-4540-b1a2-a48a13e29756",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer questions with short factoid answers.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context Str: contains relevant facts\n",
      "Query Str: ${query_str}\n",
      "Answer: often between 1 and 5 words\n",
      "\n",
      "---\n",
      "\n",
      "Context Str: What I Worked On February 2021 Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep. The first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights. The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer. I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't know enough math to do anything interesting of that type. So I'm not surprised I can't remember any programs I wrote, because they can't have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn't. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager's expression made clear. With microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping. [1] The first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer. Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he'd write 2 pages at a time and then print them out, but it was a lot better than a typewriter. Though I liked programming, I didn't plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored. I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI. AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we'd have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words. I didn't want to drop out of grad school, but how else was I going to get out? I remember when my friend Robert Morris got kicked out of Cornell for writing the internet worm of 1988, I was envious that he'd found such a spectacular way to get out of grad school. Then one day in April 1990 a crack appeared in the wall. I ran into professor Cheatham and he asked if I was far enough along to graduate that June. I didn't have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing parts of On Lisp where I could, and I was able to respond, with no perceptible delay \"Yes, I think so. I'll give you something to read in a few days.\" I picked applications of continuations as the topic. In retrospect I should have written about macros and embedded languages. There's a whole world there that's barely been explored. But all I wanted was to get out of grad school, and my rapidly written dissertation sufficed, just barely. Meanwhile I was applying to art schools. I applied to two: RISD in the US, and the Accademia di Belli Arti in Florence, which, because it was the oldest art school, I imagined would be good. RISD accepted me, and I never heard back from the Accademia, so off to Providence I went. I'd applied for the BFA program at RISD, which meant in effect that I had to go to college again. This was not as strange as it sounds, because I was only 25, and art schools are full of people of different ages. RISD counted me as a transfer sophomore and said I had to do the foundation that summer. The foundation means the classes that everyone has to take in fundamental subjects like drawing, color, and design. Toward the end of the summer I got a big surprise: a letter from the Accademia, which had been delayed because they'd sent it to Cambridge England instead of Cambridge Massachusetts, inviting me to take the entrance exam in Florence that fall. This was now only weeks away. My nice landlady let me leave my stuff in her attic. I had some money saved from consulting work I'd done in grad school; there was probably enough to last a year if I lived cheaply. Now all I had to do was learn Italian. Only stranieri (foreigners) had to take this entrance exam. In retrospect it may well have been a way of excluding them, because there were so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don't know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as high as I could to make the most of my limited vocabulary. [2] I'm only up to age 25 and already there are such conspicuous patterns. Here I was, yet again about to attend some august institution in the hopes of learning about some prestigious subject, and yet again about to be disappointed. The students and faculty in the painting department at the Accademia were the nicest people you could imagine, but they had long since arrived at an arrangement whereby the students wouldn't require the faculty to teach anything, and in return the faculty wouldn't require the students to learn anything. And at the same time all involved would adhere outwardly to the conventions of a 19th century atelier. We actually had one of those little stoves, fed with kindling, that you see in 19th century studio paintings, and a nude model sitting as close to it as possible without getting burned. Except hardly anyone else painted her besides me. The rest of the students spent their time chatting or occasionally trying to imitate things they'd seen in American art magazines. Our model turned out to live just down the street from me. She made a living from a combination of modelling and making fakes for a local antique dealer. She'd copy an obscure old painting out of a book, and then he'd take the copy and maltreat it to make it look old. [3] While I was a student at the Accademia I started painting still lives in my bedroom at night. These paintings were tiny, because the room was, and because I painted them on leftover scraps of canvas, which was all I could afford at the time.\n",
      "Query Str: What did the author do growing up?\n",
      "Answer: wrote short stories, programmed on an IBM 1401 in 9th grade, built a microcomputer from a kit, wrote simple games and a word processor on a TRS-80, studied philosophy in college, switched to AI, and painted still lives at the Accademia di Belli Arti\n",
      "\n",
      "---\n",
      "\n",
      "Context Str: YC was different from other kinds of work I've done. Instead of deciding for myself what to work on, the problems came to me. Every 6 months there was a new batch of startups, and their problems, whatever they were, became our problems. It was very engaging work, because their problems were quite varied, and the good founders were very effective. If you were trying to learn the most you could about startups in the shortest possible time, you couldn't have picked a better way to do it. There were parts of the job I didn't like. Disputes between cofounders, figuring out when people were lying to us, fighting with people who maltreated the startups, and so on. But I worked hard even at the parts I didn't like. I was haunted by something Kevin Hale once said about companies: \"No one works harder than the boss.\" He meant it both descriptively and prescriptively, and it was the second part that scared me. I wanted YC to be good, so if how hard I worked set the upper bound on how hard everyone else worked, I'd better work very hard. One day in 2010, when he was visiting California for interviews, Robert Morris did something astonishing: he offered me unsolicited advice. I can only remember him doing that once before. One day at Viaweb, when I was bent over double from a kidney stone, he suggested that it would be a good idea for him to take me to the hospital. That was what it took for Rtm to offer unsolicited advice. So I remember his exact words very clearly. \"You know,\" he said, \"you should make sure Y Combinator isn't the last cool thing you do.\" At the time I didn't understand what he meant, but gradually it dawned on me that he was saying I should quit. This seemed strange advice, because YC was doing great. But if there was one thing rarer than Rtm offering advice, it was Rtm being wrong. So this set me thinking. It was true that on my current trajectory, YC would be the last thing I did, because it was only taking up more of my attention. It had already eaten Arc, and was in the process of eating essays too. Either YC was my life's work or I'd have to leave eventually. And it wasn't, so I would. In the summer of 2012 my mother had a stroke, and the cause turned out to be a blood clot caused by colon cancer. The stroke destroyed her balance, and she was put in a nursing home, but she really wanted to get out of it and back to her house, and my sister and I were determined to help her do it. I used to fly up to Oregon to visit her regularly, and I had a lot of time to think on those flights. On one of them I realized I was ready to hand YC over to someone else. I asked Jessica if she wanted to be president, but she didn't, so we decided we'd try to recruit Sam Altman. We talked to Robert and Trevor and we agreed to make it a complete changing of the guard. Up till that point YC had been controlled by the original LLC we four had started. But we wanted YC to last for a long time, and to do that it couldn't be controlled by the founders. So if Sam said yes, we'd let him reorganize YC. Robert and I would retire, and Jessica and Trevor would become ordinary partners. When we asked Sam if he wanted to be president of YC, initially he said no. He wanted to start a startup to make nuclear reactors. But I kept at it, and in October 2013 he finally agreed. We decided he'd take over starting with the winter 2014 batch. For the rest of 2013 I left running YC more and more to Sam, partly so he could learn the job, and partly because I was focused on my mother, whose cancer had returned. She died on January 15, 2014. We knew this was coming, but it was still hard when it did. I kept working on YC till March, to help get that batch of startups through Demo Day, then I checked out pretty completely. (I still talk to alumni and to new startups working on things I'm interested in, but that only takes a few hours a week.) What should I do next? Rtm's advice hadn't included anything about that. I wanted to do something completely different, so I decided I'd paint. I wanted to see how good I could get if I really focused on it. So the day after I stopped working on YC, I started painting. I don't think it was entirely luck that the first batch was so good. You had to be pretty bold to sign up for a weird thing like the Summer Founders Program instead of a summer job at a legit place like Microsoft or Goldman Sachs. The deal for startups was based on a combination of the deal we did with Julian ($10k for 10%) and what Robert said MIT grad students got for the summer ($6k). We invested $6k per founder, which in the typical two-founder case was $12k, in return for 6%. That had to be fair, because it was twice as good as the deal we ourselves had taken. Plus that first summer, which was really hot, Jessica brought the founders free air conditioners. [16] Fairly quickly I realized that we had stumbled upon the way to scale startup funding. Funding startups in batches was more convenient for us, because it meant we could do things for a lot of startups at once, but being part of a batch was better for the startups too. It solved one of the biggest problems faced by founders: the isolation. Now you not only had colleagues, but colleagues who understood the problems you were facing and could tell you how they were solving them. As YC grew, we started to notice other advantages of scale. The alumni became a tight community, dedicated to helping one another, and especially the current batch, whose shoes they remembered being in. We also noticed that the startups were becoming one another's customers. We used to refer jokingly to the \"YC GDP,\" but as YC grows this becomes less and less of a joke. Now lots of startups get their initial set of customers almost entirely from among their batchmates. I had not originally intended YC to be a full-time job. I was going to do three things: hack, write essays, and work on YC. As YC grew, and I grew more excited about it, it started to take up a lot more than a third of my attention. But for the first few years I was still able to work on other things. In the summer of 2006, Robert and I started working on a new version of Arc. This one was reasonably fast, because it was compiled into Scheme. To test this new Arc, I wrote Hacker News in it. It was originally meant to be a news aggregator for startup founders and was called Startup News, but after a few months I got tired of reading about nothing but startups. Plus it wasn't startup founders we wanted to reach. It was future startup founders. So I changed the name to Hacker News and the topic to whatever engaged one's intellectual curiosity. HN was no doubt good for YC, but it was also by far the biggest source of stress for me. If all I'd had to do was select and help founders, life would have been so easy. And that implies that HN was a mistake. Surely the biggest source of stress in one's work should at least be something close to the core of the work. Whereas I was like someone who was in pain while running a marathon not from the exertion of running, but because I had a blister from an ill-fitting shoe. When I was dealing with some urgent problem during YC, there was about a 60% chance it had to do with HN, and a 40% chance it had do with everything else combined. [17] As well as HN, I wrote all of YC's internal software in Arc. But while I continued to work a good deal in Arc, I gradually stopped working on Arc, partly because I didn't have time to, and partly because it was a lot less attractive to mess around with the language now that we had all this infrastructure depending on it. So now my three projects were reduced to two: writing essays and working on YC. YC was different from other kinds of work I've done. Instead of deciding for myself what to work on, the problems came to me. Every 6 months there was a new batch of startups, and their problems, whatever they were, became our problems. It was very engaging work, because their problems were quite varied, and the good founders were very effective. If you were trying to learn the most you could about startups in the shortest possible time, you couldn't have picked a better way to do it. There were parts of the job I didn't like. Disputes between cofounders, figuring out when people were lying to us, fighting with people who maltreated the startups, and so on. But I worked hard even at the parts I didn't like. I was haunted by something Kevin Hale once said about companies: \"No one works harder than the boss.\"\n",
      "Query Str: What did the author do during his time at YC?\n",
      "Answer: Worked with startups, solved their problems, wrote essays, and developed internal software.\n",
      "\n",
      "---\n",
      "\n",
      "Context Str: this is my context\n",
      "Query Str: hello?\n",
      "Answer:\n"
     ]
    }
   ],
   "source": [
    "print(qa_prompt_tmpl.format(query_str=\"hello?\", context_str=\"this is my context\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f3097d9c-89e3-4d89-ae8d-6a57bd4fda0b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "query_engine = index.as_query_engine(\n",
    "    text_qa_template=qa_prompt_tmpl\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "df7b8d58-ddb0-468c-854f-bf475134d653",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = query_engine.query(\"what did the author do at RISD?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2a93b21d-2766-404e-833f-5a1cb3bf8891",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What did the author do after dropping out of RISD?\n",
      "Answer: Moved to New York, painted, and wrote a book on Lisp.\n"
     ]
    }
   ],
   "source": [
    "print(str(response))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama_index_v3",
   "language": "python",
   "name": "llama_index_v3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
